[D] How LLMs are changing search
submitted by /u/firef1y1
[link] [comments]
( 8
min )
-
Open
-
Open
-
Open
-
Open
-
Open
-
Open
Can large language models replace humans in the systematic review process? Evaluating GPT-4's efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages. (arXiv:2310.17526v1 [cs.CL])
Systematic reviews are vital for guiding practice, research, and policy, yet
they are often slow and labour-intensive. Large language models (LLMs) could
offer a way to speed up and automate systematic reviews, but their performance
in such tasks has not been comprehensively evaluated against humans, and no
study has tested GPT-4, the biggest LLM so far. This pre-registered study
evaluates GPT-4's capability in title/abstract screening, full-text review, and
data extraction across various literature types and languages using a
'human-out-of-the-loop' approach. Although GPT-4 had accuracy on par with human
performance in most tasks, results were skewed by chance agreement and dataset
imbalance. After adjusting for these, there was a moderate level of performance
for data extraction, and - barring studies that used highly reliable prompts -
screening performance levelled at none to moderate for different stages and
languages. When screening full-text literature using highly reliable prompts,
GPT-4's performance was 'almost perfect.' Penalising GPT-4 for missing key
studies using highly reliable prompts improved its performance even more. Our
findings indicate that, currently, substantial caution should be used if LLMs
are being used to conduct systematic reviews, but suggest that, for certain
systematic review tasks delivered under reliable prompts, LLMs can rival human
performance.
( 3
min )
Improved Best-of-Both-Worlds Guarantees for Multi-Armed Bandits: FTRL with General Regularizers and Multiple Optimal Arms. (arXiv:2302.13534v2 [cs.LG] UPDATED)
We study the problem of designing adaptive multi-armed bandit algorithms that
perform optimally in both the stochastic setting and the adversarial setting
simultaneously (often known as a best-of-both-world guarantee). A line of
recent works shows that when configured and analyzed properly, the
Follow-the-Regularized-Leader (FTRL) algorithm, originally designed for the
adversarial setting, can in fact optimally adapt to the stochastic setting as
well. Such results, however, critically rely on an assumption that there exists
one unique optimal arm. Recently, Ito (2021) took the first step to remove such
an undesirable uniqueness assumption for one particular FTRL algorithm with the
$\frac{1}{2}$-Tsallis entropy regularizer. In this work, we significantly
improve and generalize this result, showing that uniqueness is unnecessary for
FTRL with a broad family of regularizers and a new learning rate schedule. For
some regularizers, our regret bounds also improve upon prior results even when
uniqueness holds. We further provide an application of our results to the
decoupled exploration and exploitation problem, demonstrating that our
techniques are broadly applicable.
( 3
min )
Towards Better Generalization with Flexible Representation of Multi-Module Graph Neural Networks. (arXiv:2209.06589v4 [cs.LG] UPDATED)
Graph neural networks (GNNs) have become compelling models designed to
perform learning and inference on graph-structured data. However, little work
has been done to understand the fundamental limitations of GNNs for scaling to
larger graphs and generalizing to out-of-distribution (OOD) inputs. In this
paper, we use a random graph generator to systematically investigate how the
graph size and structural properties affect the predictive performance of GNNs.
We present specific evidence that the average node degree is a key feature in
determining whether GNNs can generalize to unseen graphs, and that the use of
multiple node update functions can improve the generalization performance of
GNNs when dealing with graphs of multimodal degree distributions. Accordingly,
we propose a multi-module GNN framework that allows the network to adapt
flexibly to new graphs by generalizing a single canonical nonlinear
transformation over aggregated inputs. Our results show that the multi-module
GNNs improve the OOD generalization on a variety of inference tasks in the
direction of diverse structural features.
( 2
min )
On the Convergence to a Global Solution of Shuffling-Type Gradient Algorithms. (arXiv:2206.05869v2 [cs.LG] UPDATED)
Stochastic gradient descent (SGD) algorithm is the method of choice in many
machine learning tasks thanks to its scalability and efficiency in dealing with
large-scale problems. In this paper, we focus on the shuffling version of SGD
which matches the mainstream practical heuristics. We show the convergence to a
global solution of shuffling SGD for a class of non-convex functions under
over-parameterized settings. Our analysis employs more relaxed non-convex
assumptions than previous literature. Nevertheless, we maintain the desired
computational complexity as shuffling SGD has achieved in the general convex
setting.
( 2
min )
Characterizing the Implicit Bias of Regularized SGD in Rank Minimization. (arXiv:2206.05794v6 [cs.LG] UPDATED)
We study the bias of Stochastic Gradient Descent (SGD) to learn low-rank
weight matrices when training deep neural networks. Our results show that
training neural networks with mini-batch SGD and weight decay causes a bias
towards rank minimization over the weight matrices. Specifically, we show, both
theoretically and empirically, that this bias is more pronounced when using
smaller batch sizes, higher learning rates, or increased weight decay.
Additionally, we predict and observe empirically that weight decay is necessary
to achieve this bias. Unlike previous literature, our analysis does not rely on
assumptions about the data, convergence, or optimality of the weight matrices
and applies to a wide range of neural network architectures of any width or
depth. Finally, we empirically investigate the connection between this bias and
generalization, finding that it has a marginal effect on generalization.
( 2
min )
Improvement in Alzheimer's Disease MRI Images Analysis by Convolutional Neural Networks Via Topological Optimization. (arXiv:2310.16857v1 [eess.IV])
This research underscores the efficacy of Fourier topological optimization in
refining MRI imagery, thereby bolstering the classification precision of
Alzheimer's Disease through convolutional neural networks. Recognizing that MRI
scans are indispensable for neurological assessments, but frequently grapple
with issues like blurriness and contrast irregularities, the deployment of
Fourier topological optimization offered enhanced delineation of brain
structures, ameliorated noise, and superior contrast. The applied techniques
prioritized boundary enhancement, contrast and brightness adjustments, and
overall image lucidity. Employing CNN architectures VGG16, ResNet50,
InceptionV3, and Xception, the post-optimization analysis revealed a marked
elevation in performance. Conclusively, the amalgamation of Fourier topological
optimization with CNNs delineates a promising trajectory for the nuanced
classification of Alzheimer's Disease, portending a transformative impact on
its diagnostic paradigms.
( 2
min )
Improving Few-shot Generalization of Safety Classifiers via Data Augmented Parameter-Efficient Fine-Tuning. (arXiv:2310.16959v1 [cs.LG])
As large language models (LLMs) are widely adopted, new safety issues and
policies emerge, to which existing safety classifiers do not generalize well.
If we have only observed a few examples of violations of a new safety rule, how
can we build a classifier to detect violations? In this paper, we study the
novel setting of domain-generalized few-shot learning for LLM-based text safety
classifiers. Unlike prior few-shot work, these new safety issues can be hard to
uncover and we do not get to choose the few examples. We demonstrate that
existing few-shot techniques do not perform well in this setting, and rather we
propose to do parameter-efficient fine-tuning (PEFT) combined with augmenting
training data based on similar examples in prior existing rules. We empirically
show that our approach of similarity-based data-augmentation + prompt-tuning
(DAPT) consistently outperforms baselines that either do not rely on data
augmentation or on PEFT by 7-17% F1 score in the Social Chemistry moral
judgement and 9-13% AUC in the Toxicity detection tasks, even when the new rule
is loosely correlated with existing ones.
( 2
min )
-
Open
Cause-Effect Inference in Location-Scale Noise Models: Maximum Likelihood vs. Independence Testing. (arXiv:2301.12930v3 [cs.LG] UPDATED)
A fundamental problem of causal discovery is cause-effect inference, learning
the correct causal direction between two random variables. Significant progress
has been made through modelling the effect as a function of its cause and a
noise term, which allows us to leverage assumptions about the generating
function class. The recently introduced heteroscedastic location-scale noise
functional models (LSNMs) combine expressive power with identifiability
guarantees. LSNM model selection based on maximizing likelihood achieves
state-of-the-art accuracy, when the noise distributions are correctly
specified. However, through an extensive empirical evaluation, we demonstrate
that the accuracy deteriorates sharply when the form of the noise distribution
is misspecified by the user. Our analysis shows that the failure occurs mainly
when the conditional variance in the anti-causal direction is smaller than that
in the causal direction. As an alternative, we find that causal model selection
through residual independence testing is much more robust to noise
misspecification and misleading conditional variance.
( 2
min )
Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theory. (arXiv:2307.04204v2 [cs.LG] UPDATED)
Cohen et al. (2021) empirically study the evolution of the largest eigenvalue
of the loss Hessian, also known as sharpness, along the gradient descent (GD)
trajectory and observe the Edge of Stability (EoS) phenomenon. The sharpness
increases at the early phase of training (referred to as progressive
sharpening), and eventually saturates close to the threshold of $2 /
\text{(step size)}$. In this paper, we start by demonstrating through empirical
studies that when the EoS phenomenon occurs, different GD trajectories (after a
proper reparameterization) align on a specific bifurcation diagram independent
of initialization. We then rigorously prove this trajectory alignment
phenomenon for a two-layer fully-connected linear network and a single-neuron
nonlinear network trained with a single data point. Our trajectory alignment
analysis establishes both progressive sharpening and EoS phenomena,
encompassing and extending recent findings in the literature.
( 2
min )
Improved Best-of-Both-Worlds Guarantees for Multi-Armed Bandits: FTRL with General Regularizers and Multiple Optimal Arms. (arXiv:2302.13534v2 [cs.LG] UPDATED)
We study the problem of designing adaptive multi-armed bandit algorithms that
perform optimally in both the stochastic setting and the adversarial setting
simultaneously (often known as a best-of-both-world guarantee). A line of
recent works shows that when configured and analyzed properly, the
Follow-the-Regularized-Leader (FTRL) algorithm, originally designed for the
adversarial setting, can in fact optimally adapt to the stochastic setting as
well. Such results, however, critically rely on an assumption that there exists
one unique optimal arm. Recently, Ito (2021) took the first step to remove such
an undesirable uniqueness assumption for one particular FTRL algorithm with the
$\frac{1}{2}$-Tsallis entropy regularizer. In this work, we significantly
improve and generalize this result, showing that uniqueness is unnecessary for
FTRL with a broad family of regularizers and a new learning rate schedule. For
some regularizers, our regret bounds also improve upon prior results even when
uniqueness holds. We further provide an application of our results to the
decoupled exploration and exploitation problem, demonstrating that our
techniques are broadly applicable.
( 3
min )
Hierarchical clustering with OWA-based linkages, the Lance-Williams formula, and dendrogram inversions. (arXiv:2303.05683v2 [stat.ML] UPDATED)
Agglomerative hierarchical clustering based on Ordered Weighted Averaging
(OWA) operators not only generalises the single, complete, and average
linkages, but also includes intercluster distances based on a few nearest or
farthest neighbours, trimmed and winsorised means of pairwise point
similarities, amongst many others. We explore the relationships between the
famous Lance-Williams update formula and the extended OWA-based linkages with
weights generated via infinite coefficient sequences. Furthermore, we provide
some conditions for the weight generators to guarantee the resulting
dendrograms to be free from unaesthetic inversions.
( 2
min )
Mutual information of spin systems from autoregressive neural networks. (arXiv:2304.13412v2 [cond-mat.stat-mech] UPDATED)
We describe a new direct method to estimate bipartite mutual information of a
classical spin system based on Monte Carlo sampling enhanced by autoregressive
neural networks. It allows studying arbitrary geometries of subsystems and can
be generalized to classical field theories. We demonstrate it on the Ising
model for four partitionings, including a multiply-connected even-odd division.
We show that the area law is satisfied for temperatures away from the critical
temperature: the constant term is universal, whereas the proportionality
coefficient is different for the even-odd partitioning.
( 2
min )
Curvature Filtrations for Graph Generative Model Evaluation. (arXiv:2301.12906v3 [cs.LG] UPDATED)
Graph generative model evaluation necessitates understanding differences
between graphs on the distributional level. This entails being able to harness
salient attributes of graphs in an efficient manner. Curvature constitutes one
such property that has recently proved its utility in characterising graphs.
Its expressive properties, stability, and practical utility in model evaluation
remain largely unexplored, however. We combine graph curvature descriptors with
emerging methods from topological data analysis to obtain robust, expressive
descriptors for evaluating graph generative models.
( 2
min )
The statistical thermodynamics of generative diffusion models. (arXiv:2310.17467v1 [stat.ML])
Generative diffusion models have achieved spectacular performance in many
areas of generative modeling. While the fundamental ideas behind these models
come from non-equilibrium physics, in this paper we show that many aspects of
these models can be understood using the tools of equilibrium statistical
mechanics. Using this reformulation, we show that generative diffusion models
undergo second-order phase transitions corresponding to symmetry breaking
phenomena. We argue that this lead to a form of instability that lies at the
heart of their generative capabilities and that can be described by a set of
mean field critical exponents. We conclude by analyzing recent work connecting
diffusion models and associative memory networks in view of the thermodynamic
formulations.
( 2
min )
Squared Neural Families: A New Class of Tractable Density Models. (arXiv:2305.13552v2 [cs.LG] UPDATED)
Flexible models for probability distributions are an essential ingredient in
many machine learning tasks. We develop and investigate a new class of
probability distributions, which we call a Squared Neural Family (SNEFY),
formed by squaring the 2-norm of a neural network and normalising it with
respect to a base measure. Following the reasoning similar to the well
established connections between infinitely wide neural networks and Gaussian
processes, we show that SNEFYs admit closed form normalising constants in many
cases of interest, thereby resulting in flexible yet fully tractable density
models. SNEFYs strictly generalise classical exponential families, are closed
under conditioning, and have tractable marginal distributions. Their utility is
illustrated on a variety of density estimation, conditional density estimation,
and density estimation with missing data tasks.
( 2
min )
Improving Neural Additive Models with Bayesian Principles. (arXiv:2305.16905v2 [stat.ML] UPDATED)
Neural additive models (NAMs) can improve the interpretability of deep neural
networks by handling input features in separate additive sub-networks. However,
they lack inherent mechanisms that provide calibrated uncertainties and enable
selection of relevant features and interactions. Approaching NAMs from a
Bayesian perspective, we enhance them in three primary ways, namely by a)
providing credible intervals for the individual additive sub-networks; b)
estimating the marginal likelihood to perform an implicit selection of features
via an empirical Bayes procedure; and c) enabling a ranking of feature pairs as
candidates for second-order interaction in fine-tuned models. In particular, we
develop Laplace-approximated NAMs (LA-NAMs), which show improved empirical
performance on tabular datasets and challenging real-world medical tasks.
( 2
min )
Kernel Stein Discrepancy thinning: a theoretical perspective of pathologies and a practical fix with regularization. (arXiv:2301.13528v3 [math.ST] UPDATED)
Stein thinning is a promising algorithm proposed by (Riabiz et al., 2022) for
post-processing outputs of Markov chain Monte Carlo (MCMC). The main principle
is to greedily minimize the kernelized Stein discrepancy (KSD), which only
requires the gradient of the log-target distribution, and is thus well-suited
for Bayesian inference. The main advantages of Stein thinning are the automatic
remove of the burn-in period, the correction of the bias introduced by recent
MCMC algorithms, and the asymptotic properties of convergence towards the
target distribution. Nevertheless, Stein thinning suffers from several
empirical pathologies, which may result in poor approximations, as observed in
the literature. In this article, we conduct a theoretical analysis of these
pathologies, to clearly identify the mechanisms at stake, and suggest improved
strategies. Then, we introduce the regularized Stein thinning algorithm to
alleviate the identified pathologies. Finally, theoretical guarantees and
extensive experiments show the high efficiency of the proposed algorithm. An
implementation of regularized Stein thinning as the kernax library in python
and JAX is available at https://gitlab.com/drti/kernax.
( 3
min )
Approximate Leave-one-out Cross Validation for Regression with $\ell_1$ Regularizers (extended version). (arXiv:2310.17629v1 [math.ST])
The out-of-sample error (OO) is the main quantity of interest in risk
estimation and model selection. Leave-one-out cross validation (LO) offers a
(nearly) distribution-free yet computationally demanding approach to estimate
OO. Recent theoretical work showed that approximate leave-one-out cross
validation (ALO) is a computationally efficient and statistically reliable
estimate of LO (and OO) for generalized linear models with differentiable
regularizers. For problems involving non-differentiable regularizers, despite
significant empirical evidence, the theoretical understanding of ALO's error
remains unknown. In this paper, we present a novel theory for a wide class of
problems in the generalized linear model family with non-differentiable
regularizers. We bound the error |ALO - LO| in terms of intuitive metrics such
as the size of leave-i-out perturbations in active sets, sample size n, number
of features p and regularization parameters. As a consequence, for the
$\ell_1$-regularized problems, we show that |ALO - LO| goes to zero as p goes
to infinity while n/p and SNR are fixed and bounded.
( 2
min )
Risk-Averse Model Uncertainty for Distributionally Robust Safe Reinforcement Learning. (arXiv:2301.12593v2 [cs.LG] UPDATED)
Many real-world domains require safe decision making in uncertain
environments. In this work, we introduce a deep reinforcement learning
framework for approaching this important problem. We consider a distribution
over transition models, and apply a risk-averse perspective towards model
uncertainty through the use of coherent distortion risk measures. We provide
robustness guarantees for this framework by showing it is equivalent to a
specific class of distributionally robust safe reinforcement learning problems.
Unlike existing approaches to robustness in deep reinforcement learning,
however, our formulation does not involve minimax optimization. This leads to
an efficient, model-free implementation of our approach that only requires
standard data collection from a single training environment. In experiments on
continuous control tasks with safety constraints, we demonstrate that our
framework produces robust performance and safety at deployment time across a
range of perturbed test environments.
( 2
min )
Characterizing the Implicit Bias of Regularized SGD in Rank Minimization. (arXiv:2206.05794v6 [cs.LG] UPDATED)
We study the bias of Stochastic Gradient Descent (SGD) to learn low-rank
weight matrices when training deep neural networks. Our results show that
training neural networks with mini-batch SGD and weight decay causes a bias
towards rank minimization over the weight matrices. Specifically, we show, both
theoretically and empirically, that this bias is more pronounced when using
smaller batch sizes, higher learning rates, or increased weight decay.
Additionally, we predict and observe empirically that weight decay is necessary
to achieve this bias. Unlike previous literature, our analysis does not rely on
assumptions about the data, convergence, or optimality of the weight matrices
and applies to a wide range of neural network architectures of any width or
depth. Finally, we empirically investigate the connection between this bias and
generalization, finding that it has a marginal effect on generalization.
( 2
min )
